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ABSTRACT 

This  document  presents  approaches  for  assessing  the  impact  of  Information  Technology  on 
Command  and  Control.  The  determination  of  the  contribution  of  Command,  Control, 
Communications  and  Intelligence  (C3I)  systems  on  military  mission  effectiveness  has  proven  to  be 
a  complex  problem,  particularly  with  the  depth  and  breadth  of  military  organizations,  and  the  wide 
range  of  their  operations.  The  evaluation  of  C3I  systems  necessitates  the  application  of  a  framework 
and  a  methodology  to  yield  appropriate  Measures  of  Merit.  No  single  measure  or  methodology 
exists  that  satisfactorily  assesses  the  overall  effectiveness  of  command  and  control  systems. 
Therefore  a  multi-method,  multi -phase  approach  is  necessary.  Hierarchies  of  measures  are  defined, 
example  frameworks  and  structures  for  assessments  are  presented,  and  factors  relevant  to  and 
affecting  these  assessments  are  discussed. 


RESUME 

Ce  document  decrit  des  methodes  pour  evaluer  Timpact  de  la  technologie  de  I'information 
sur  le  commandement  et  controle.  L' Evaluation  de  la  contribution  des  systemes  de  commandement, 
controle,  communications  et  renseignements  (C3R)  a  Tefficacite  des  missions  militaires  est  une 
question  complexe.  Cette  complexite  augment  avec  Tampeur  et  la  profondeur  des  organisations 
militaires,  ainsi  que  la  diversite  des  opErations  auxquelles  elles  sont  participent.  L' Evaluation  de 
C3R  nEcessite  Fapplication  d'un  cadre  de  travail  et  d'une  mEthodologie  permettant  Faeces  aux 
mesures  d'efficacitE  appropriEes.  Aucune  approche  ni  mEthodologie  ne  permet  k  Fheure  actuelle 
d' Evaluer  de  fa9on  appropriEe  FefficacitE  globale  des  systemes  de  commandement  et  controle.  Par 
consEquent,  une  approche  par  phases  impliquant  plusieurs  mEthodes  est  nEcessaire.  Un  ensemble 
de  mesures  hiErarchiques  est  dEfini,  des  exemples  de  cadres  de  travail  et  de  structures  d' Evaluation 
sont  prEsentEs  et  les  facteurs  pertinents  qui  pouraient  affecter  ces  Evaluations  sont  discutEs. 
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EXECUTIVE  SUMMARY 

The  determination  of  the  impact  of  Command,  Control,  Communications  and  Intelligence 
(C3I)  systems  on  military  mission  effectiveness  has  proven  to  be  a  complex  problem,  particularly 
with  the  depth  and  breadth  of  military  organizations,  and  the  wide  range  of  military  operations.  The 
key  considerations  of  any  assessment  are  the  establishment  of  the  objectives  in  the  evaluation,  the 
definition  of  what  is  meant  by  beneficial,  in  what  maimer,  and  by  whom.  The  benefits  should  be 
evaluated  by  the  impact  on  the  fulfilment  of  the  military  objectives  within  the  scope  of  potential 
missions.  The  impact  should  be  measured  in  terms  of  defined  qualities  that  are  relevant  to  the 
objectives,  and  must  span  the  hierarchies  of  the  military  organization. 

The  benefits  of  the  application  of  information  technologies  (IT)  on  C3I  must  be  determined 
in  a  reliable  and  systematic  manner.  The  benefits  attributed  to  information  systems  may  be 
statistically  difficult  to  prove  with  the  multitude  of  uncontrollable  variables  and  unmeasurable 
attributes.  From  a  system’s  perspective,  the  classical  approach  is  to  compare  the  output  of  the 
system  with  its  input.  Measures  of  productivity  are  defined  in  terms  of  the  ratio  of  output  to  input, 
which  are  multidimensional  and  problematic  to  identify.  In  this  context,  the  system  encompasses 
the  total  military  environment,  with  IT  contributing  to  the  processes  within  the  system.  The 
incremental  contribution  of  IT  would  be  reflected  by  an  incremental  improvement  in  productivity 
and  therefore  effectiveness. 

A  four  level  hierarchy  of  measures  has  been  proposed  by  the  US  Military  Operational 
Research  Society.  They  encompass:  a)  Measures  of  Force  Effectiveness  (MOFE)  for  measures  of 
mission  effectiveness;  b)  Measures  of  Effectiveness  (MOE)  for  measures  of  functional 
organizational  performance  external  to  C3I  systems  ;  c)  Measures  of  Performance  (MOP)  for 
measures  of  attributes  of  internal  system  behaviour;  and  d)  Dimensional  Parameters  (DP)  for 
measures  of  characteristics  inherent  in  physical  entities. 

No  single  measure  or  methodology  exists  that  satisfactorily  assesses  the  overall  effectiveness 
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of  C3I  systems;  therefore  a  multi-method,  multi-phase  approach  is  necessary.  The  evaluation  of  C3I 
systems  necessitates  the  application  of  a  framework  and  a  methodology  to  yield  appropriate 
Measures  of  Merit  (MOM).  Analyses  of  C3I  systems  reveal  a  complex  hierarchical  composition. 
A  structured  resolution/functional  decomposition  approach  can  be  related  to  the  organizational 
structure  to  yield  performance  measures  for  the  organization  as  a  whole,  for  individual  components 
within  the  organization,  and  for  specific  tasks  within  the  cells.  An  evaluation  framework 
encompasses  several  factors:  a)  the  evaluation  configuration;  b)  the  evaluation  goal  or  purpose,  c) 
the  context,  assumptions,  and  constraints,  d)  the  evaluation  domain,  e)  the  identification  of  specific 
measures,  f)  the  specification  of  the  measures,  g)  the  scenario  or  stimulus,  and  h)  the  collection 
means. 

Other  factors  relevant  for  evaluation  include  the  role  of  Command  and  Control  models,  scenarios, 
and  the  effects  of  uncertainty. 
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1.0  INTRODUCTION 

Throughout  history,  the  application  of  state-of-the-art  technologies  has  been  a  force  multiplier 
for  military  effectiveness.  The  Gulf  War  graphically  portrayed  the  impact  of  technology  on  weapon 
effectiveness,  where  targets  were  destroyed  with  unprecedented  precision.  However,  it  is  less  evident 
how  technology  contributes  to  the  broader  scope  of  Command  and  Control  (C2). 

The  determination  of  the  impact  of  Command,  Control,  Communications  and  Intelligence 
(C3I)'  systems  on  military  mission  effectiveness  has  proven  to  be  a  complex  problem,  particularly 
with  the  depth  and  breadth  of  military  organizations  eind  the  wide  range  of  military  operations. 
Improved  C3I  system  performance  does  not  necessarily  imply  enhanced  effectiveness  of  military 
operations.  It  is  often  difficult  to  show  convincingly  that  investments  in  modem  information 
technology  for  command  and  control  may  be  more  beneficial  than  investments  in  other  areas.  The 
Navy  is  currently  upgrading  their  C3I  system  (MCOIN),  the  Army  is  procuring  a  system  (LFCS),  the 
Air  Force  is  in  the  process  of  defining  a  system  (AFCCIS),  and  a  high  level  joint  system  is  also  in  the 
definition  stage.  Occasionally,  decisions  to  invest  in  systems  are  made  without  necessarily  following 
a  rigorous  analysis  of  the  potential  benefits.  The  Auditor  General’s  report  of  1994  [Ref.  1]  stated 
that  the  Information  Technology  (IT)^  systems  should  be  subject  to  a  full  cost-benefit  analysis  with 
quantified  performance  factors. 

The  key  considerations  of  any  assessment  are  the  establishment  of  the  evaluation  objectives, 
the  definition  of  what  is  meant  by  beneficial,  in  what  manner,  and  by  whom.  The  benefits  should 
be  evaluated  by  the  impact  on  the  fulfilment  of  the  military  objectives  within  the  scope  of  potential 
missions.  The  impact  should  be  measured  in  terms  of  defined  qualities  that  are  relevant  to  the 


'  The  nuances  between  the  definitions  of  C3I  and  Command  and  Control  (C2)  are  not 
discussed  in  this  document  and  may  be  used  interchangeably. 

^  The  application  of  Information  Technology  implies  Command  and  Control  Information 
Systems. 
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objectives,  and  must  span  the  hierarchies  of  military  organization. 

During  the  last  two  decades  many  new  automated  command  and  control  systems  have  been 
developed  and  fielded.  However,  progress  in  the  determination  of  the  effectiveness  of  these  systems 
has  been  limited.  The  US  Military  Operations  Research  Society  (MORS)  has  sponsored  several 
workshops  on  Measures  of  Merit  (MOM)  since  1985.  These  workshops  have  led  to  the  development 
of  an  analysis  framework  (MCES  -  Modular  Command  and  Control  Evaluation  Structure)  for  the 
measurement  of  performance  and  effectiveness  within  a  conceptual  model  for  Command  and  Control 
[Ref.  5,  23].  The  US  TRADOC  Analysis  Center  (TRAC)  developed  the  C2  Measures  of 
Effectiveness  Handbook  [Ref.  2]  in  1990  based  on  the  MORS  workshops. 

The  NATO  AC/243  Panel  7  Ad  Hoc  Working  Group  (AHWG)  on  the  Impact  of  C3I  on  the 
Battlefield  [Ref.  20]  acknowledged  that  the  specification  of  measures  of  effectiveness  is  difficult. 
The  1992  final  report  recommended  that  a  hierarchy  of  measures  be  established  as  an  important  step 
in  understanding  overall  system  effectiveness,  and  that  systems  be  analyzed  at  different  levels  of 
detail.  The  types  of  measures  were  grouped  relating  to  C3I  system’s  performance,  force/commander 
effectiveness,  and  battle  outcome.  To  quote  from  the  final  report,  “Measures  ...  are  often 
inadequate  and  too  model  or  scenario  specific.  In  addition,  they  have  often  been  generated  in  ad  hoc 
ways,  suggesting  a  lack  of  formal  analysis  in  their  development.” 

In  1995,  NATO  Research  Study  Group  19  was  formed  with  the  aim  of  identifying 
methodologies  for  doing  C3I  evaluations  in  order  to  provide  information  on  how  to  represent  C3I 
effects  on  military  operations.  The  product  of  the  RSG-19  was  a  Code  of  Best  Practice  (COBP)  for 
the  determination  of  effectiveness  of  C3I,  with  emphasis  on  models.  Command  and  Control 
organizations,  scenarios,  and  measures  of  evaluation  [Ref.  21].  This  document  reflects  in  part  the 
author’s  contribution  to  the  COBP,  i.e.  the  Measures  of  Merit  section  for  which  he  was  responsible; 
members  of  RSG-19  also  provided  valuable  suggestions  on  measures  of  merit  for  C3I. 
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2.0  MEASURES  OF  MERIT 


2.1  Background 

The  benefits  of  the  application  of  information  technologies  for  C3I  must  be  determined  in  a 
reliable  and  systematic  manner.  With  hundreds  or  even  thousands  of  variables  and  attributes  involved 
in  complex  Command  and  Control  systems,  and  vvith  boundary  conditions  that  are  often  not  specified, 
this  goal  is  a  particularly  challenging  one.  The  benefits  attributed  to  information  systems  may  be 
statistically  difficult  to  prove  ■with  the  multitude  of  uncontrollable  variables  and  unmeasurable 
attributes.  From  a  system’s  perspective,  the  classical  approach  is  to  compare  the  output  of  a  system 
with  its  input.  Measures  of  productivity  are  defined  in  terms  of  the  ratio  of  output  to  input,  which 
are  multidimensional  and  problematic  to  identify.  In  this  context,  the  system  encompasses  the  total 
military  environment,  with  IT  contributing  to  the  processes  within  the  system.  The  incremental 
contribution  of  IT  would  be  reflected  by  an  incremental  improvement  in  productivity  and  therefore 
effectiveness. 

Information  systems  are  expensive  to  procure,  deploy  and  maintain;  the  value  of  these  systems 
is  expected  to  overcome  the  costs.  The  task  remains  of  quantifying  the  term  value,  and  of  identifying 
linkages  to  the  effectiveness  of  the  organization.  To  enable  this  identification,  the  effectiveness  of 
information  systems  requires  the  a  priori  identification  of  the  purpose  of  the  system.  By  adopting  a 
process-oriented  view  of  the  organization,  measurements  may  be  identified  from  the  impact  of 
information  technology  on  intermediate  organizational  processes.  Benefits  such  as  cost  reduction  and 
increased  productivity  may  be  discemable.  Nevertheless,  the  benefits  of  IT  may  be  intangible  (for 
example,  increased  flexibility)  and  are  therefore  difficult  to  identify,  quantify  and  measure.  IT  may 
also  provide  value-added  attributes  of  increased  precision  and  decreased  response  times,  i.e.,  more 
accurate  information  with  quicker  response  times  would  provide  better  decisions  within  limited 
windows  of  opportunity. 
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In  a  business  environment,  financial  indicators  such  as  Return  On  Investment,  Return  On 
Equity,  Return  On  Assets,  etc.  are  often  used  as  outputs.  One  may  expect  a  relationship  such  as 
depicted  in  Fig.  1,  with  continuing  improvement  in  return  for  increasing  investment.  However,  the 
relationship  to  an  input,  such  as  investment  in  technological  assets,  to  these  indicators  is  not 
obvious.  Some  researchers,  such  as  Strassmann  [Ref  22],  raise  the  spectre  of  the  productivity 
paradox  with  the  claim  that  there  is  no  relationship  between  investments  in  information  technology 
and  productivity  (see  Fig.  2,  which  plots  investments  from  different  companies).  This  apparent 
paradox  is  controversial,  since  the  data  plots  the  results  of  the  performance  of  different  unrelated 
companies  to  the  performance  indicators.  A  more  appropriate  approach  is  to  measure  the  incremental 
improvement  in  productivity  with  incremental  investments  in  technology  within  the  same 
organization. 
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FIGURE  2  -  Cost  /  Benefit  Samples 


Financial  indicators  of  the  business  community  may  not  be  applicable  in  the  military 
environment,  and  an  added  perspective  on  measures  of  the  benefits  of  information  technology  at  the 
high  level  is  required.  From  the  system’s  perspective,  the  military  “output”  may  be  defined  in  terms 
of  peacetime  readiness  or  force  effectiveness.  The  determination  of  the  impact  of  IT  in  turn  requires 
the  functional  decomposition  through  the  organization’s  hierarchies  to  identify  military  processes 
affected  by  IT. 

2.2  Purposes  of  Obtaining  Evaluation  Measures 

An  important  step  in  the  design  of  an  evaluation  is  the  establishment  of  the  objectives  of  the 
evaluation.  With  clear  objectives,  the  formulation  and  identification  of  the  measurements  is  greatly 
facilitated.  Measures  of  Merit  (MOM)  have  different  purposes  across  a  wide  scope,  with  specific 
intentions,  as  outlined  below: 

a.  for  new  requirements,  the  establishment  of  standards  or  expectations  of  performance; 
the  establishment  of  the  bounds  of  performance  of  a  system  and  the  effects  of  imposed 


constraints; 
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b.  the  comparison  and  selection  of  alternative  systems  that  may  be  very  different  but  are 
designed  to  achieve  a  similar  purpose; 

c.  the  assessment  of  the  utilization  of  a  system  in  new  or  unexpected  application  domains  or 
missions; 

d.  the  identification  of  potential  weaknesses  in  specific  areas  of  an  organization  or  system 
(areas  of  high  error  potential  or  high  user  workload); 

e.  the  analysis  of  the  impact  of  organisational  change; 

f.  the  determination  of  the  effectiveness  of  training; 

g.  the  determination  of  the  most  cost-effective  approaches  to  achieve  desired  objectives; 

h.  the  comparison  of  a  replacement  system,  or  components  of  the  system,  against  its 
predecessor; 

i.  assistance  in  requirements  generation  and  validation,  and  the  derivation  of  specific 
Command  and  Control  requirements  from  broad  statements  of  objectives; 

j.  the  evaluation  of  the  effectiveness  and  performance  of  human  decision  making  in  the 
Command  and  Control  cycle. 

The  clear  statement  of  the  objectives  of  an  evaluation  allows  one  to  focus  on  the  definition 
of  measures,  and  to  plan  the  means  of  obtaining  values  for  these  measures. 
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2.3  Definitions  of  Measures 

The  important  issues  raised  by  decision  makers  are  how  to  arrive  at  judgments  on  the  degree 
that  Command  and  Control  acquisitions  may  improve  force  effectiveness,  and  subsequently  provide 
convincing  demonstrations  of  the  improvements.  An  approach  may  be  to  define  a  single  measure  that 
objectively  reflects  the  military  and  political  objectives  of  the  missions  under  consideration. 
However,  such  a  measure  may  not  be  practically  obtainable,  and  furthermore,  a  single  measure  would 
not  provide  a  means  of  analyzing  the  impact  of  specific  command  and  control  elements  on  force 
effectiveness. 

To  understand  the  impact  of  C3I,  it  is  not  only  necessary  to  analyze  and  measure  the  effect 
of  the  C3I  on  military  operations,  but  also  of  the  components  of  the  constituent  systems.  Therefore, 
classes  of  measures  are  required,  with  appropriate  definitions.  The  terms  “effectiveness”  and 
“performance”  may  be  viewed  as  the  same  concept,  but  viewed  fi'om  different  perspectives,  i.e. 
“performance”  may  be  viewed  in  terms  of  “effectiveness”  at  a  higher  level.  These  expressions  are 
often  used  interchangeably,  with  ambiguous  reference  to  what  are  the  objects  of  measure.  The  term 
Measures  of  Merit  is  recommended  as  a  generic  terminology  to  encompass  different  measures.  The 
measures  are  best  defined  in  hierarchical  levels  related  to  each  other,  each  concerning  its  own 
boundary.  From  the  conceptual  viewpoint,  the  level  of  analysis  and  the  context  in  which  the 
measurements  are  made  are  important. 

Within  the  MCES  framework,  MORS  has  developed  a  four-level  hierarchy  of  measures  from 
the  high  level  of  force  effectiveness  to  the  low  level  of  rudimentary  measures  of  physical  entities. 
These  are: 
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a.  Measures  of  Force  Effectiveness  (MOFE):  Measures  of  how  a  military  force  achieves  its 
mission  or  the  degree  it  meets  its  objectives.  Examples  are  loss  exchange  ratios,  number 
of  targets  destroyed. 

b  Measures  of  Effectiveness  (MOE):  Measures  that  are  external  to  C3I  systems  of 
functional  organizational  performance  within  the  operational  context.  Examples  are 
number  of  targets  detected,  time  to  perform  specific  tasks. 

c.  Measures  of  Performance  (MOP):  Measures  of  attributes  of  internal  system  behaviour. 
Examples  are  information  retrieval  times,  message  transmission  times. 

d.  Dimensional  Parameters  (DP).  Properties  or  characteristics  inherent  in  the  physical 
entities,  e.g.  signal  to  noise  ratios,  bandwidth. 

This  hierarchical  definition  allows  the  linking  of  lower  level  measures  to  higher  levels.  The 
lower  level  measures  may  be  analyzed  to  provide  an  indication  of  the  impact  on  higher  level 
measures.  However,  the  linkages  may  be  difficult  to  establish,  the  assumptions  and  constraints  must 
be  clearly  stated  and  critical  paths  at  the  same  level  must  be  identified.  The  effectiveness  of  C3I 
systems  must  be  evaluated  in  terms  of  their  influence  on  military  effectiveness,  with  the  appropriate 
determination  of  the  linkages  and  causal  relationships  of  the  measures  of  merit. 

The  relationships  between  MOM,  the  environment,  forces,  C2  systems  and  their  components 
are  presented  in  a  simplified  form  in  Fig.  3.  For  a  particular  analysis,  it  is  important  to  identify  the 
specific  boundaries,  whereupon  the  measures  are  then  bound  to  the  specific  context. 
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Environment 


FIGURE  3  -  MORS  Relationships  of  Measures  of  Merit 

2.4  Characteristics  of  Measures 

The  relationship  between  these  types  of  measures  is  often  difficult  to  establish  authoritatively. 
Linkages  between  lower  level  measures  (DP  and  MOP)  are  easier  to  determine  than  those  to  the 
higher  level  measures  (MOE  and  MOFE),  particularly  when  small  numbers  of  observations  are 
available,  so  that  correlation  techniques  are  not  appropriate  and/or  controls  are  not  available  to  isolate 
causal  effects.  Higher  level  measures  tend  to  be  more  subjective  while  lower  level  measures  tend  to 
be  more  quantitative.  MOE  and  MOFE  are  more  dependent  on  the  operational  context  and  therefore 
more  scenario  dependent.  This  implies  the  necessity  for  a  selected  range  of  scenarios  for  the  proper 
analysis  and  interpretation  of  the  measures.  MOFE  tend  to  be  few  in  number  and  expensive  to  obtain. 
By  themselves  MOFE  do  not  reveal  much  about  the  underlaying  C3I  system,  and  therefore  should 
be  used  in  context  with  MOE  and  MOP  that  provide  “diagnostic”  information  about  the  dynamics 
of  the  C3I  process.  The  two  central  levels  (MOE  and  MOP)  yield  the  most  insight  into  command  and 
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control  systems.  DP  should  be  collected  when  they  can  be  obtained  cost  effectively  in  order  to  check 
for  physical  characteristics  of  the  C3I  system  components  that  may  be  creating  problems  at  the  MOP 
level.  Table  I  lists  general  characteristics  of  the  types  of  measures. 


TABLET 

Tendeneies  of  Charaeteristics  of  MOM 


MOM 

Scenario 

Cost/ 

Effort 

Number 

Value 

Compre¬ 

hension 

Subj/ 

Obj 

Type 

MOFE 

MOE 

MOP 

Depen¬ 

dent 

High 

Few 

Less 

(alone) 

Military 

Sub¬ 

jective 

outcome 

DP 

Indepen¬ 

dent 

Low 

Many 

More 

Techni¬ 

cal 

Ob¬ 

jective 

process 

2.5  Properties 

This  section  outlines  measurement  theory  concepts  that  apply  to  ensure  that  the  right 
measuring  instruments  are  selected  and  applied  correctly.  By  definition,  measurement  is  the 
assignment  of  values  to  observation  units  expressing  properties  of  the  units.  Three  levels  of  measures 
relate  numbers  to  properties  of  interest:  ordering  in  rank  (e.g.,  small  to  large),  equal  intervals  (e.g., 
50%  inerease  in  baud  rate),  and  ratios  or  absolute  values  (e.g.,  56  kilobaud).  A  fourth  level  may  be 
defined  as  nominal  assignment,  i.e.,  using  numbers  as  labels,  but  is  not  relevant  to  MOM.  Ratio  level 
measurement  possesses  an  absolute  zero  value,  and  has  the  properties  of  interval  level  measurements. 
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In  turn,  interval  level  measures  possess  both  quantity  and  ordering. 

The  key  properties  for  quality  assurance  in  evaluation  methodology  are  reliability  and  validity. 
A  third  property  is  cost  and  convenience;  the  measurement  should  be  cost-effective  and  easy  to  apply. 
This  property  bears  no  direct  relationship  with  validity,  but  reliability  is  directly  related  to  cost. 
Reliable  measurements  require  repeated  measurements  to  obtain  large  sample  sizes.  A  cost-effective 
measurement  plan  provides  just  enough  data  for  useful  and  definitive  conclusions.  However,  cost 
may  be  an  overriding  factor  in  system  evaluation. 

Failure  to  take  into  account  the  principles  of  validity  and  reliability  raises  the  risk  of 
generating  false  conclusions.  Validity  and  reliability  are  not  absolutes,  but  matters  of  degree. 
Validity  is  the  degree  to  which  the  measure  focuses  on  the  attribute  of  interest  and  only  on  that 
attribute.  Reliability  represents  precision  and  repeatability.  A  measure  may  be  reliable  but  not  valid, 
or  it  may  be  valid  but  not  reliable.  In  order  to  link  the  performance  of  a  system  as  a  whole  against  that 
of  its  components,  the  measures  must  correspond  to  critical  tasks  to  be  valid. 

MCES  lists  several  quality  criteria  for  measures,  which  may  be  grouped  according  to  the 
properties  related  to  reliability  and  validity  according  to  the  following  tables. 
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TABLE n 

MCES  Reliability  Criteria  of  Measures 


Reliability  Criteria 

Definition 

Discriminatory 

Measurable 

Quantitative 

Objective 

Sensitive 

Can  identify  real  differences  between  alternatives 

Can  be  computed  or  estimated 

Can  be  assigned  numbers  or  ranks 

Can  be  defined  or  derived,  independent  of  subjective  opinion 

Can  reflect  changes  in  system  variables 

TABLE  m 

MCES  Validity  Criteria  of  Meastires 


Validity  Criteria 

Definition 

Mission  Oriented 

Relate  to  force/system  mission 

Realistic 

Relate  realistically  to  the  C2  system  and  associated  uncertainties 

Appropriate 

Relate  to  acceptable  standards  and  analysis  objectives 

Inclusive 

Reflect  those  standards  required  by  the  analysis  objectives 

Simple 

Easily  understood  by  users 

2.5.1  Validity 

The  property  of  validity  may  be  categorized  into  four  types:  internal,  construct,  statistical  and 
external. 


Internal  validity  is  defined  as  the  establishment  of  causal  relationships  between  variables  of 
interest.  This  is  necessary  to  validate  the  hypothesis  that  a  given  measure  is  responsible  for  a  specific 
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effect  on  another  measure. 

Construct  (also  referred  to  as  content)  validity  means  that  the  target  obj  ect,  and  only  the  obj  ect, 
is  measured. 

Statistical  validity  implies  that  sufficient  sensitivity  is  involved  in  order  to  determine 
relationships  between  independent  and  dependent  variables.  Statistical  tests  control  two  types  of 
errors  in  measurement.  Type  I  or  alpha  is  the  probability  of  rejecting  a  claimed  hypothesis  while  it 
is  true  (e.g.,  concluding  that  a  new  automated  decision  aid  does  not  reduce  the  time  to  prepare  a  plan, 
while  it  in  fact  does).  Type  II  or  beta  is  the  probability  of  accepting  a  hypothesis  while  it  is  not  true 
(e.g.,  concluding  that  a  new  decision  aid  does  reduce  planning  times,  when  in  fact  it  does  not). 

External  validity  is  the  extent  to  which  results  may  be  extended  to  other  populations  or 
environments.  Associated  with  this  is  expert  validity,  which  refers  to  the  degree  to  which  measures 
are  accepted  by  those  who  are  knowledgeable  in  the  field. 

2.5.2  Reliability 

Reliability  involves  the  expectation  of  errors  associated  with  measurements.  It  is  defined  as 
the  amount  of  measurement  precision,  as  measured  by  the  variance  of  repeated  measurements  of  the 
same  object.  The  key  principles  of  reliability  are  consistency  (repeatability)  and  precision.  The 
amount  of  error  associated  with  measurements  must  be  known  to  interpret  the  results  and  to 
discriminate  between  the  real  effects  and  error  effects.  Reliability  assures  that  variance  in  a  measure 
between  systems  is  due  to  system  differences  and  not  errors  in  measurements.  Reliability  cannot  be 
calculated,  but  only  estimated.  The  correlation  between  two  observations  of  the  same  measure  is  an 
estimate  of  the  reliability.  Theoretically,  the  validity  correlation  caimot  exceed  the  square  root  of  the 
estimated  reliability  of  the  measure.  The  variance  in  measurements  may  be  due  to  random  errors 
and/or  systematic  errors.  Systematic  errors  may  be  reduced  by  applying  different  measurement 
approaches,  and  random  ones  by  repetitive  measurements. 
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2.6  Types  of  Measures 

A  common  thread  in  the  approaches  for  evaluation  is  the  functional  decomposition  of  the  C3I 
cycle.  Command  and  control  effectiveness  therefore  depends  upon  the  functional  processes  of  the 
command  and  control  system,  and  the  evaluation  of  functions  may  be  determined  by  data  measured 
at  the  task  level. 

The  evaluation  of  tasks  provides  the  most  detailed  insight  into  C3I  activities.  The  primary 
measures  are  in  terms  of  time  consumed  and  accuracy  of  actions.  Task  analysis  must  be  performed 
prior  to  evaluation,  with  the  identification  of  the  task  definition  and  of  the  critical  elements  for 
successful  task  completion. 

Performance  measures  of  a  system’s  behaviour  may  thus  be  reduced  to  measures  based  on 
time,  accuracy,  or  a  combination  of  factors  which  may  be  interdependent.  Time-based  measures  are 
quantitative,  while  accuracy  measures  may  also  be  subjective. 

2.6.1  Time 

For  command  and  control  tasks,  time-based  metrics  include: 

1)  Time  taken  to  react  to  an  event  (time  to  notice  new  information,  recognize/classify 

information,  understand  information); 

2)  Time  to  perform  a  task  (time  to  make  a  decision); 

3)  Time  into  the  future  for  predictive  analysis;  and 


4)  Rate  of  performing  tasks  (tempo). 
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2.6.2  Accuracy 

Metrics  for  accuracy  include: 

a.  precision  of  performance; 

b.  reliability  of  performance; 

c.  completeness  (known  unknowns,  unknown  unknowns); 

d.  errors  (alpha,  beta,  omission,  commission,  transposition,  severity);  and 

e.  quality  of  decisions. 

Some  accuracy  measures  may  be  assessed  in  units  of  time;  for  example,  the  time  taken  to 
detect  an  error.  The  quality  of  decisions  is  difficult  to  evaluate  objectively,  except  by  focussing  on 
outcomes.  The  processes  involved  may  have  to  be  examined  to  obtain  objective  measures,  and 
subject  matter  experts  may  be  called  upon  to  make  an  evaluation.  Accuracy  of  information  implies 
both  the  accuracy  of  the  data  and  the  accuracy  of  the  interpretation  of  the  data. 


2.6.3  Relationships  -  time  and  accuracy 

Time  based  and  accuracy  based  measures  often  bear  an  inverse  relationship,  implying  a 
tradeoffbetween  speed  of  performance  and  accuracy  of  performance.  Speed  of  performance  must  be 
specified  in  terms  of  minimum  desired  accuracy  or  completeness,  and  accuracy  measurements  in 
terms  of  time  available.  Therefore,  the  specification  of  thresholds  or  standards  for  metrics  must  be 
referenced  in  terms  of  imposed  constraints. 
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2.7  Examples  of  Hierarchical  Measures 

This  section  outlines  examples  of  Measures  of  Merit.  Since,  as  previously  stated,  MOE  and 
MOP  provide  the  most  insight  into  C2,  emphasis  is  put  on  these  hierarchical  levels. 

2.7.1  Measures  of  Force  Effectiveness  -  MOFE 

MOFE  are  high  level  measures  that  are  few  in  number  with  more  military  than  technical  meaning. 
Examples  of  such  measures  include: 

a.  territory  gained  /  lost; 

b.  mission  objectives  achieved; 

c.  battle  won  /  lost; 

d.  ratio  of  enemy  losses  to  friendly  losses; 

e.  number  of  targets  destroyed;  £ind 

f.  percent  Combat  Effectiveness. 

2.7.2  Measures  of  Effectiveness -MOE 

Table  IV  contains  examples  of  time-  and  accuracy-based  measures.  These  measures  are  of  the 
MOE  level.  The  measures  may  be  expressed  in  quantative  units  of  time,  percentage,  or  absolute 
values,  or  of  subjective  interpretation. 
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TABLE  IV 

MOE  Time  and  Accuracy  Measures 


Measure 

Example 

Time 

Based 

Time  taken  to  perform  a  fixed  task  or 

sequence  of  tasks 

Planning  tasks 

Time  to  perform  a  variable  task 

Developing  and  selecting  options  or  courses 

of  action 

Time  to  recognize  or  respond  to  an  event 

Response  to  a  critical  enemy  contact 

Time  to  achieve  a  target  state 

Tactical  objective 

Percentage  of  time  on  target 

Enemy  location  data  up  to  date 

Intelligence  latency 

Time  taken  to  reflect  ground  truth 

Number  of  events  in  queue 

Messages  pending  action 

Timeliness  of  response 

Fire  plan  schedule 

Accuracy 

Based 

Accuracy  or  precision  of  performance  of  tasks 

Information  on  maps,  data  bases 

Sensitivity  of  detecting  system  events 

Recognition  of  events  requiring  change  in 

plans 

Probability  of  making  errors 

Errors  in  fire  plan  target  schedules 

Time  to  recognize  existence  of  error 

Necessity  for  plan  alteration 

Time  to  recover  from  error 

Time  to  redo  part  of  plan 

Knowledge  of  current  system  status 

Comprehension  of  battle  situation 

Quality  of  decision  making 

Quality  of  tactical  plan 
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2.7.3  Measures  of  Performance -MOP 

A  bottom-up  approach  [Ref.  16]  to  the  establishment  of  measures  provides  three  types: 
technical  services  attributes,  user  effectiveness  attributes,  and  ineffectiveness  attributes.  These  are 
outlined  in  the  following  table,  and  yield  measures  mainly  at  the  MOP  level. 


TABLE  V 
MOP  Measures 


Technical  Services  Attributes  -  Hardware  and  Software 

Availability 

Fimctional  capabilities  available  to  users 

Survivability 

Ability  to  survive  partial  destruction  of  system 

Robustness/Endurance 

Ability  to  adapt  to  enviromnent 

Maintainability 

Ease  of  repair  or  replacement  during  operation 

Computation  Capacity 

Acceptable  response  times  to  users 

Portability 

Ability  to  operate  on  different  platforms 

Mobility 

Ability  to  move  with  operational  units 

Technical  Services  -  Applications  Attributes 

Interoperability 

Communications  with  other  C3I  systems 

Security 

Confidentiality  and  integrity  of  data 

Confidentiality 

Information  protected  at  appropriate  level 

Integrity 

Required  for  confidence  of  data 

Customizability 

Ability  to  customize  parameters  to  actual  activities 

Quantity  of  Information 

Provide  all  information  required  by  user 

Bandwidth 

Ability  to  support  multi-media 
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User  Effectiveness  -  Information  Quality 

Selectivity 

Ability  to  provide  required  information  at  required  amount 

Accuracy 

Degree  information  provided  is  correct 

Comprehension 

Facilitate  understanding  of  situation 

User  Effectiveness  -  Time  Related 

Response  time 

Response  to  requests  within  established  times 

Timeliness 

Information  available  at  appropriate  time 

Ease  of  use 

Ease  of  access  to  information  in  a  timely  manner 

Training  time 

Time  to  train  users 

Decision  response  time 

Time  available  to  commanders 

User  Effectiveness  -  New  Capability  Related  Attributes 

Uncertainty  Management 

Ability  to  take  into  accoimt  ambiguity  and  uncertainty 

Simulation 

Ability  to  realistically  test  courses  of  action  before  decision 

Information  Age  Management 

Ability  to  store  information  for  later  analyses 

User  Effectiveness  -  Motivation 

Communication  Acknowledgment 

Positive  feedback 

Knowledge  of  Consequences 

Positive  for  stimulus  but  negative  for  stress 

Confidence  of  Information 

Not  used  if  untrustworthy 

User  friendliness 

Adaptability  to  different  users 

Participation  in  Development 

User  buy-in 
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Ineffectiveness  Attributes 

Loneliness 

Isolation  from  real  world 

Interference 

Micro  management 

Wait  and  see 

Waiting  for  confirmation  instead  of  deciding 

Wargame 

C3I  reflects  real  situations 

2.7.4  Dimensional  Parameters 

At  this  level,  the  focus  is  on  subsystem  parameters  which  are  usually  static  and  provide 
information  on  the  capabilities  of  equipment.  Examples  are; 

a.  Signal-to-noise  ratios 

b.  Bandwidth 

c.  Resolution 

d.  Floating  point  operations  per  second 

e.  Radar  cross-section 

f.  Radar  resolution 

g.  Bit  error  rate. 


2.8  Summary  of  Measures 

It  has  been  stated  that  MOE  and  MOP  provide  the  most  insight  to  C2.  These  measures 
described  in  the  above  sections,  among  others,  are  summarized  in  Table  VI. 
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Table  VI 

Summary  of  Measures 


Measure 

Definition 

Time 

Elapsed  duration 

Accuracy 

Degree  of  correctness 

Completeness 

Extent  known  of  whole  or  desired  objective  attained 

Cost 

Savings  in  operations,  maintenance 

Adaptability 

Anticipation  of  contingencies 

Flexibility 

Ability  to  respond  to  foreseen  possibilities 

Robustness 

Ability  to  withstand  unforeseen  situations 

Sustainability 

Ability  to  maintain  tempo 

Survivability 

Ability  to  withstand  degradation 

S  ecurityA^  ulnerability 

Information  Operations 

Reliability/Availability 

Extent  system  performs  as  expected 

Efficiency 

Reduced  manpower  and  resource  requirements 

Readiness/Responsiveness 

Delays  in  achieving  desired  state 

Navigation 

Mechanisms  to  obtain  information,  data  structures 

Coherence 

Logical  and  consistent  relationships 

Consistency 

Same  results  with  same  conditions  and  actions 

Sensitivity 

Degree  variations  in  input  required  to  effect  outcomes 

Integrity 

Soundness  of  operations 

Relevance 

Appropriate  reference 

Risk  reduction 

Reduced  possibility  of  negative  outcomes 

Confidence 

Reduced  uncertainty 
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2.9  Headquarters  C&C  Performance  Measures 

Performance  measures  may  be  divided  to  sets  corresponding  to  the  C2  cycle: 

a.  Monitor:  information  transmission,  values,  times,  effect,  comprehension; 

b.  Plan:  information  exchange,  impact,  alteration,  quality;  and 

c.  Direct  and  disseminate. 

The  measures  are  related  to  three  levels:  the  headquarters  as  a  whole,  the  individual  cells 
within  the  headquarters,  and  the  performance  of  specific  tasks  within  the  cells. 

Appendix  A  outlines  an  approach  for  land  forces  Division  HQ  evaluations  for  planning  and 
monitoring  [Ref.  11]. 
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3.0  EVALUATION  FRAMEWORK 

The  evaluation  of  C3I  systems  necessitates  the  application  of  a  framework  and  methodology 
to  yield  appropriate  Measures  of  Merit.  Analyses  of  C3I  systems  reveal  a  complex  hierarchical 
composition.  A  structured  resolution/functional  decomposition  approach  can  be  related  to  the 
organizational  structure  to  yield  performance  measures  for  the  organization  as  a  whole,  individual 
components  within  the  organization,  and  specific  tasks  within  the  cells. 

Assuming  that  Command  and  Control  effectiveness  is  synonymous  with  overall  unit 
effectiveness,  MOM  could  be  obtained  by  addressing  the  outcomes  or  products  of  unit  activities. 
Goal  level  evaluation  attempts  to  define  the  ability  of  the  specific  formation  to  make  the 
environmental  state  match  the  goal  (directive)  provided  by  the  superior  headquarters.  The  degree  that 
the  environmental  state  matches  the  desired  goal  states  indicates  a  level  of  effectiveness. 
Alternatively,  Command  and  Control  effectiveness  may  be  viewed  as  dependent  on  the  functional 
processes  of  the  command  and  control  system,  with  measures  obtained  mainly  at  the  task  level. 

An  evaluation  framework  encompasses  several  factors: 

a.  the  evaluation  configuration  (e.g.,  storyboard,  testbed,  full-scale  simulation,  field  trial); 

b.  the  evaluation  goal  or  purpose; 

c.  the  context,  assumptions,  and  constraints; 

d.  the  evaluation  domain,  i.e.,  MOFE,  MOE,  MOP,  DP; 

e.  the  identification  of  specific  measures; 
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f.  the  specification  of  the  measures; 

g.  the  scenario  or  stimulus; 

h.  the  collection  means,  e.g.,  subject  matter  experts,  automatic  logging;  and 

i.  the  interpretation  of  the  results; 


3.1  Modular  Command  And  Control  Evaluation  Structure  (MCES) 

MCES  prescribes  a  process  of  measurement,  but  does  not  identify  either  a  measurement 
system  or  a  set  of  measures.  Similarly,  while  calling  for  the  collection  of  data,  MCES  does  not 
provide  details  on  how  data  are  to  be  collected.  MCES  does  provide  guidance  on  how  good  measures 
and  good  collection  procedures  are  characterized,  but  leaves  the  details  of  the  measurement,  data 
collection,  and  analysis  plans  to  the  analyst. 

Evaluation  of  C3I  effectiveness  requires  a  comprehensive  approach  for  the  preparation  of  the 
evaluation  process,  the  collection  of  data,  and  its  interpretation.  MCES  addresses  both  the 
managerial  and  analytical  aspects  of  evaluation,  and  was  originally  developed  for  the  systematic 
comparison  of  C2systems.  The  objective  of  MCES  is  to  guide  analysts  in  the  identification  of 
appropriate  measures  for  estimating  the  effect  of  C2  on  combat. 

MCES  considers  C2  as  consisting  of  three  components:  physical  entities  (equipment,  software., 
people),  structure  (interrelationships  between  entities)  and  processes  (C2  functions).  The  boundary 
of  a  C2  system  is  defined  as  a  delineation  between  the  system  studied  and  the  environment.  The 
TRADOC  C2MOE  Handbook  adds  mission  objective  as  the  top  layer  of  the  hierarchy  of  C2 


components. 
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MCES  focuses  on  measures  as  opposed  to  models,  but  includes  the  cybernetic  loop  (Lawson) 
model  of  generic  C3.  It  consists  of  seven  procedural  steps  as  shown  in  Table  VII 


Table  VII 
MCES  Steps 


Step 

Procedure 

Product 

1 

Formulate  the  problem 

Problem  Statement  &  Scenario 

2 

Bound  C2  system 

System  Elements 

3 

Define  C2  process 

System  Functions 

4 

Integrate  elements  and  functions 

System  Architecture 

5 

Specify  measures  at  boundaries 

Fimctional  Measures 

6 

Generate  data  for  measures 

Values 

7 

Aggregate  and  integrate  results. 

Analysis  of  Results 

3.1.1  Problem  Formulation 

This  step  provides  a  description  of  the  analysis  obj  ectives  from  the  viewpoint  of  the  C2  system 
life  cycle,  and  the  level  of  analysis  prescribed.  Assumptions  are  stated,  and  the  scenarios  selected. 

3.1.2  C2  System  Bounding 

The  system  elements  are  identified  and  categorized.  The  C2  system  of  interest  is  bound  in 
terms  ofphysical  entities  (equipment,  software,  staff),  structure  (organization,  concepts  of  operation), 
and  C2  processes.  The  boundaries  are  identified  in  terms  of  subsystems,  C2  systems,  own  forces. 
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and  the  environment. 

3.1.3  C2  Process  Definition 

The  functions  of  the  C2  process  are  identified  and  mapped  to  the  generic  C2  cycle  (sense, 
assess,  generate,  select,  plan,  direct),  and  the  inputs  and  outputs  to  the  processes  are  identified. 
Insight  to  C2  is  obtained  by  decomposing  the  C2  cycle  into  functions/subfunctiohs,  processes,  and 
tasks.  These  are  situation  independent  descriptions  of  responsibilities  of  the  elements  constituting 
the  military  organization.  The  Monitor,  Assess,  Plan,  and  Direct  high  level  functions  are  further 
decomposed  into  sub-functions.  The  term  processes  in  this  context  defines  the  interrelationships  of 
tasks  performed  to  fulfill  the  functions. 

3.1.4  Integration  of  Elements  and  Functions 

This  step  defines  the  relationships  between  the  C2  processes,  physical  entities,  and  structure. 
Input/output  relations  are  derived  to  describe  the  internal  information  flows  between  separate  process 
functions,  and  physical  entities  that  perform  functions  are  mapped  to  the  outputs.  The  hierarchical 
relationships  between  C2  functions  are  determined,  and  an  organizational  structure  is  produced. 

3.1.5  Specification  of  Measures 

This  step  results  in  the  specification  of  measures  focussed  primarily  on  the  C2  process 
functions.  The  measures  are  classified  according  to  the  four  levels  previously  defined.  The  MCES 
defines  the  following  attributes  for  MOM: 

a.  name  and  category; 

b.  system  reference  (boundary); 

c.  function  reference  (purpose); 

d.  units  of  measure; 
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e.  value  measured;  and 

f.  threshold  value. 


3.1.6  Data  Generation 

The  generation  of  values  for  the  identified  measures  may  be  performed  by  various  means  such 
as  exercises,  experiments,  simulations,  and  subjective  judgments.  The  values  may  be  measured 
directly  or  be  derived  from  other  measures.  For  the  evaluation  of  new  systems,  the  generation  may 
be  assisted  by  designing  into  the  system  functions  to  yield  specific  measurements. 

3.1.7  Aggregation  of  Measures 

For  the  aggregation  and  analysis,  particular  attention  must  be  paid  on  the  causality, 
sufficiency,  and  independence  of  measures.  Some  measures  (e.g.,  time  intervals  in  the  C2  cycle)  may 
be  directly  aggregated,  although  the  critical  paths  must  be  identified  for  meaningful  results. 
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4.0  EXAMPLE  METHODOLOGIES 
4.1  Army  Command  and  Control  Evaluation  System  (ACCES) 

ACCES  [Ref.  4,  12-15]  is  a  derivation  of  HEAT  (Headquarters  Effectiveness  Assessment 
Tool)  [Ref  3,  6-10],  which  was  developed  primarily  for  joint  theatre  level  operations.  ACCES 
reorganized  HEAT  concepts  into  army  doctrinal  language  and  doctrine,  but  shares  the  same 
philosophy.  ACCES  has  been  applied  to  numerous  Division  and  Corps  command  centre  assessments. 
It  represents  a  comprehensive  set  of  practical  and  objective  performance  measurements  for  C2 
activities.  The  primary  focus  of  ACCES  is  the  overall  performance  of  a  command  centre,  or  network 
of  command  centres,  at  various  stages  of  the  conmiand  and  control  process,  from  the  collection  of 
data  to  the  conversion  of  data  into  intelligence,  and  to  the  implementation  of  plans  and  directives. 
The  underlying  approach  to  ACCES  is  that  command  and  control  comprises  interdependent  sub¬ 
process  that  can  be  observed  and  measured.  ACCES  considers  C2  as  an  adaptive  control  process, 
where  information  collected  from  the  outside  is  processed  internally  for  generation  of  plans,  which 
may  be  adapted  to  new  information.  ACCES  takes  the  view  that  the  overall  effectiveness  of  a 
command  centre  can  be  judged  by  the  viability  of  its  plans.  A  good  plan  is  one  that  can  be  executed 
without  the  need  for  modification  beyond  the  contingencies  stated  in  the  plan,  and  remains  in  effect 
throughout  its  intended  life.  The  ACCES  measures  are  listed  in  Table  VIII. 
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Table  VIE 
ACCES  Measures 


MONITOR 

Completeness 

Commander's  Information  List  Requirements  (CILR)  for  which 

there  are  data. 

Accuracy 

Units  or  CILR  items  for  which  headquarters  data  are  within  the 

desired  window  of  accuracy. 

Querying 

Units  where  most  recent  data  are  outside  the  desired  time  window. 

Timeliness 

Units  for  which  the  most  recent  data  are  within  the  desired  time 

window. 

Impact  on  plan 

Control  cycles  initiated  because  of  monitoring  errors. 

Forecast  correctness 

Predictions  of  time  at  which  change  in  weather  and  terrain  are 

correct. 

ASSESS  AND  UNDERSTAND 

Completeness 

Periodic  briefings  requiring  an  imderstanding  of  the  situation  to  be 

expressed,  at  which  the  understanding  is  actually  stated. 

Quality 

Perceptions  of  the  situation  held  by  the  headquarters,  scored  as 

percentage  correct,  not  incorrect,  or  incorrect. 

Impact  of  plan 

Control  cycles  caused  because  headquarters  understanding  did  not 

match  ground  truth. 

Time  to  imder stand 

Time  jfrom  the  expression  of  understanding  to  the  end  of  the  period 

covered  for  the  understanding. 
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GENERATE  AND  PREDICT 

Multiple  options 

Number  of  options  considered. 

Multiple  planners 

Staff  members  who  participate  in  the  development  of  alterative 

course  of  action. 

Prediction  completeness 

Estimates  that  include,  for  each  option  presented,  predictions  of 
enemy  reaction,  degree  of  mission  accomplishment,  and  residual 
capacity  of  friendly  and  enemy  units  involved. 

Prediction  quality 

Predictions  scored  as  "correct",  "not  incorrect"  or  "incorrect". 

Prediction  time 

Time  from  the  making  of  an  estimate  to  the  end  of  the  time  covered 
by  the  associated  predictions. 

DECIDE 

Plan  quality 

Plan  assignments  that  remain  in  force  unchanged  for  the  intended 
period,  expressed  as  a  percentage  of  the  total  assignments.  The 
assignments  in  a  plan  are:  mission,  assets,  boundaries,  and 

schedules. 

Plan  congruence 

Control  cycles  arising  from  minor,  moderate,  or  major 

incongruence. 

PLAN 

Time  from  decision 

Time  taken  to  issue  a  directive  after  a  decision  has  been  made. 

Plan  consistency 

Assignments  in  implementing  directives  which  do  not  contradict  a 

Commander’s  decision. 

Plan  clarity 

Directives  not  queried  by  recipients. 

Plan  cycle  time 

Time  used  to  complete  the  control  cycle  given  minor,  moderate,  or 

major  incongruence. 
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DIRECT 

Plan  lead  time  adequacy 

Directives  for  which  planning  lead  time  provided  to  subordinates  is 

adequate. 

Time  induced  aborts 

Directives  for  which  lead  time  is  insufficient  to  allow  a  suitable 

response. 

Query  response  time 

Queries  responded  to  within  the  specified  time. 

In  addition  to  the  above  measures,  ACCES  includes  two  other  sets  of  measures:  (a) 
coordination  measures,  which  are  used  to  assess  how  well  the  staff  gather  information  within  their 
command  centre  and  with  counterparts  in  other  command  centres  and  (b)  report  measures  used  to 
assess  how  well  the  command  maintains  communications  with  superior,  subordinate,  and  adjacent 
commands. 


4.2  Multi-Attribute  Utility  Theory  (MAUT) 

Multi-Attribute  Utility  Theory  or  MAUT  [Ref  1],  is  similar  to  ACCES  in  the  sense  that  both 
use  functional  decomposition  and  function  specific  evaluation  metrics.  The  major  difference  is  that 
MAUT  assigns  a  utility  component  for  each  element  and  node  in  the  hierarchy.  MAUT  then 
aggregates  upwards  the  weighted  scores  to  provide  composite  scores  of  effectiveness. 

MAUT  has  been  used  by  some  practitioners  to  assess  C3I  in  a  particular  context.  MAUT  is 
based  on  decision  analysis  and  makes  a  variety  of  assumptions.  The  first  is  that  the  analyst  knows 
all  the  factors  that  will  influence  performance.  The  second  is  that  the  analyst  can  properly  weight 
those  factors.  Finally,  MAUT  assumes  that  the  relationships  between  the  independent  and  dependent 
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variables  are  known  and  properly  specified.  MAUT  is  therefore  like  MCES.  It  specifies  a  process 
for  analysis,  but  leaves  the  task  of  identifying  appropriate  C3I  theory  and  measures  to  the  analyst. 

The  four  key  components  of  the  MAUT  framework  are  stimulus,  hypothesis,  option,  and  response, 
which  could  be  mapped  onto  the  C3I  cycle.  These  functions  are  overlayed  into  an  evaluation 
methodology  of  three  aspects: 

a.  the  interface  between  the  system  and  user; 

b.  the  relationship  between  the  user  and  the  organization;  and 

c.  he  interface  between  the  organization  and  the  environment. 

Measures  specific  to  each  of  the  above  aspects  are  identified,  aided  by  the  decomposition  into 
a  hierarchy  of  processes,  functions,  tasks  and  activities.  For  each  element  and  node  in  the  hierarchy, 
a  utility  component  is  assigned  in  perspective  of  the  overall  evaluation  goal.  The  MAUT  approach 
does  not  consider  all  tasks  and  functions  assessed  to  be  of  equal  value  and  importance.  Finally,  the 
weighted  scores  are  aggregated  upwards  to  provide  composite  scores  for  each  nodal  function  and 
interface  domain  to  obtain  a  measure  of  overall  effectiveness. 

Ultimately,  MAUT  analyses  are  only  as  good  as  the  measurement  and  theory  underlying  them. 
For  well  understood,  narrowly  focussed  problems,  MAUT  can  be  a  powerful  tool  that  both  provides 
a  sound  structure  for  describing  relationships  and  allows  a  variety  of  sensitivity  analyses.  However, 
in  poorly  structured  problems  or  weak  knowledge  domains,  MAUT  is  simply  a  way  of  looking  at  the 
consequences  of  different  sets  of  assumptions 
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5.0  OTHER  FACTORS  FOR  EVALUATION 


5.1  C2  Models 

A  model  is  an  abstraction  representing  the  relationships  among  variable  attributes  of  a  system, 
and  incorporates  all  variables  of  interest  that  may  affect  outcomes.  Given  the  value  of  known 
attributes,  the  model  provides  predictive  capabilities  to  estimate  the  values  of  other  attributes  with 
acceptable  degrees  of  uncertainty.  Whatever  form  a  C2  model  assumes  (e.g.,  analytical,  simulation), 
values  should  be  produced  for  variables  required  to  determine  MOM  with  adequate  detail  and 
accuracy.  Models  are  limited  to  the  extent  of  the  envelope  of  possibilities  of  military  operations.  For 
example  few  models  exist  for  joint  operations  or  echelon  above  corps  and  models  may  be  limited  in 
the  scope  of  information  warfare  or  other  specific  factors. 

5.2  Scenarios 

Relevant  scenarios  are  required  to  stimulate  each  battlefield  function,  sub-functions,  tasks,  and 
organizational  components  within  the  context  of  the  environmental  aspects.  Not  all  of  these  elements 
continuously  interrelate  and  critical  tasks  must  be  identified.  The  scenario  prescribes  the  tempo  of 
operations  and  induces  specific  components  of  a  C2  system  to  react.  Partial  evaluations  may  be 
merged  by  applying  a  range  of  (possibly  overlapping)  scenarios,  with  clearly  defined  hierarchies. 
Obviously  scenarios  and  the  models  must  reflect  common  hierarchies  and  levels  of  resolution. 

5.3  Effects  of  Uncertainty 

In  order  to  state  a  level  of  confidence  in  the  interpretation  of  MOM,  the  underlaying 
assumptions  must  be  clearly  stated  and  uncertainties  be  recognized.  Uncertainties  manifest 
themselves  in  several  ways  that  may  affect  Measures  of  Merit,  and  may  be  grouped  as  follows: 

a.  Uncertainties  in  the  scenario  (model  input):  i.e.,  relevance  to  the  purpose  of  the 
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evaluation,  uncertainties  in  the  military  objective,  knowledge  of  enemy  concept  of 
operations,  intentions,  capabilities,  weapon  performance,  uncertainties  in  terrain  data; 

b.  Uncertainties  in  the  model  (structural  uncertainty):  human  performance,  parameters, 
objects,  attributes,  processes,  effects  of  constraints,  effects  of  aggregation  and 
disaggregation,  deterministic  (usually  high  level  and  low  resolution)  versus  stochastic 
models  (low  hierarchical  level  and  high  resolution); 

c.  Uncertainties  in  the  outcome:  hypersensitivity  to  input  variations,  (instability  or  chaos 
theory  where  small  variations  in  input  could  result  in  vastly  different  outcomes),  effects 
of  model  non-linearities  and  non-monotonic  behaviour  (effects  of  thresholds),  decision 
making  for  local  versus  global  optimisations. 

Sensitivity  analysis  may  be  applied  to  reduce  xmcertainty.  By  varying  the  assumptions  and 
input  data  within  ranges  of  uncertainty,  excursions  in  the  analysis  will  provide  insight  into  the  effects 
of  uncertainty. 

It  should  be  noted  that  uncertainties  in  the  scenario  often  reflect  real-world  situations  where 
a  range  of  diverse  anticipated  conditions  may  be  encoimtered,  or  even  unanticipated  conditions 
occur.  Measures  under  these  conditions  that  may  be  considered  are  flexibility  and  robustness-, 
flexibility  for  different  anticipated  situations,  and  robustness  for  the  unanticipated. 
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5.4  MOM  for  Operations  Other  Than  War 

While  national  policies  and  NATO  require  that  military  forces  be  prepared  for  high  intensity 
conflict,  the  forces  have  been  increasingly  involved  in  low  intensity  conflicts  and  Operations  Other 
Than  War  (OOTW).  OOTW  include  operations  of  force  deployment  to  create  or  maintain  conditions 
for  a  political  solution  for  disputes  before  escalation  into  hostilities.  Threats  to  international  and 
national  security  may  also  unfold  from  natural  disasters,  organized  crime,  civil  unrest,  migration, 
other  territorial  intrusions.  Most  OOTW  are  characterized  by  their  joint  or  combined  nature. 

While  the  determination  of  MOM  has  been  stated  as  difficult  to  obtain,  OOTW  offers  even 
more  of  a  challenge,  particularly  at  the  MOFE  level.  Traditional  MOE  and  MOFE  such  as  percentage 
of  time  on  target,  loss  exchange  ratios,  combat  effectiveness,  duration  of  the  campaign  are  not  always 
applicable  to  OOTW.  In  such  operations,  military  forces  may  play  important  roles,  but  political 
concerns  may  limit  the  scope  of  imposition  of  solutions.  Public  and  political  pressures  may  result  in 
shifts  in  the  selection  of  criteria  for  MOM,  e.g.,  more  emphasis  may  be  put  on  personnel  casualties 
and  less  on  equipment  losses. 

Mobility  may  be  important  for  OOTW,  as  will  as  sustainability  and  self-sufficiency  in  theatre, 
with  the  implication  of  reliability  and  maintainability.  Furthermore,  the  perception  of  the  capabilities 
of  deployed  forces  acts  as  a  deterrence  or  coercion  on  each  party  in  conflict. 

Normality  Indicators  [Ref  17]  have  been  proposed  as  an  indirect  method  for  measuring  the 
effects  of  military  involvement  in  OOTW  but  causal  relationships  are  difficult  to  prove.  These 
measures  may  be  obtained  by  evaluating  the  extent  that  conditions  have  been  restored  to  previous 
levels. 


Ref  18  developed  low-level  measures  to  cover  OOTW.  These  include,  for  example: 


UNCLASSIFIED 

36 

a.  time  between  arrival  of  friendly  forces  in  the  area  and  deployment  of  the  forces; 


b.  time  between  deployment  of  friendly  forces  and  contact  with  hostile  forces; 

c.  length  of  time  hostile  forces  were  imder  observation  without  posing  a  threat  to  friendly 
forces;  and 

d.  length  of  time  friendly  forces  are  in  potential  danger  (i.e.  hostile  forces  have  opportunity 
to  fire  on  friendly  forces). 


Ref  1 7  identified  a  structure  that  provides  high-level  measures  for  OOTW .  These  are: 

a.  opportunities  to  employ  forces,  which  reflects  the  range  of  military  capabilities  available; 

b.  strategic  deployment,  related  to  deploying  and  recovering  the  right  force  to  theatre 
efficiently  and  in  time; 

c.  endurance,  to  maintain  an  effective  force  in  theatre  for  an  extended  time; 

d.  mission  objectives,  to  measure  the  success  of  achieving  military  objectives  OOTW ;  and 

e.  successful  termination,  to  deal  with  progress  to  the  desired  end  state  (the  criteria  may  be 
political  and  not  yield  measures  related  to  military  activities). 


5.5  Measures  of  Policy  Effectiveness 

For  Operations  Other  Than  War,  political  factors  are  paramount  and  considerations  must  be 
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taken  into  account  such  as  media  coverage,  local  regional  stability,  and  sustainment  of  community 
societal  standards.  A  new  category  of  measures  may  be  added  to  characterize  the  contribution  of 
military  actions  to  broader  policy  societal  outcomes.  Measures  of  Policy  Effectiveness.  While 
military  missions  may  not  directly  achieve  policy  objectives,  they  may  provide  an  environment  more 
conducive  to  these  objectives.  However,  measures  of  effectivenesss  of  military  tasks  should  quantify 
performance  against  military  missions,  not  the  overall  policy  aspirations. 

5.6  Information  Operations 

Information  Operations  (10)  may  be  considered  as  information  dominance  over  opposing 
forces  and  its  military  application  (from  pre-  to  post-hostilities)  in  terms  of  Command  and  Control 
Warfare.  The  aims  are  to  influence,  degrade,  or  destroy  adversary  command  and  control  capabilities 
and  information  links,  while  using  countermeasures  to  protect  own  C2  assets.  10  is  a  form  of 
comprehensive  warfare  that  may  span  peacetime,  crisis,  and  combat  governed  by  a  strategy  to  obtain 
desired  objectives  that  may  be  military,  political,  or  economic.  The  ultimate  objective  is  to  affect  the 
adversary’s  decision  process,  implying  a  hierarchical  effect  on  opposing  forces  operations  with,  for 
example,  attacks  on  sensors  affecting  information  systems  to  the  propagation  of  functional  effects  on 
decision  making.  For  MOM,  hierarchical  levels  similar  to  the  MORS  may  be  appropriate,  but  the 
MOM  would  be  difficult  to  evaluate  and  become  quite  subjective  at  the  higher  levels. 
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6.0  CHALLENGES /ISSUES 

Linkage  ofDP-MOP-MOE-MOFE\  An  improvement  at  one  level  does  not  necessarily  imply 
an  improvement  at  a  higher  level,  e.g.,  increasing  target  detection  MOE  may  result  in  an  increase  rate 
of  fire  and  number  of  targets  destroyed  (MOPE),  but  only  to  the  maximum  sustainable  rate  of  fire. 
Furthermore,  the  objective  in  an  assessment  may  not  be  for  improved  MOPE,  but  for  improvement 
at  a  lower  level.  IT  may  improve  optimization  of  the  utilization  of  assets  with  reduced  resource 
requirements,  without  an  intended  effect  on  an  MOPE.  The  correlation  of  measures  with  processes 
(e.g.  battle  outcome  against  lower  level  measures)  requires  considerable  insight  into  the  C2 
environment.  The  accomplishment  of  the  aggregation  of  measures  requires  the  identification  of 
critical  paths  at  the  same  level,  the  discrimination  of  measures  and  the  examination  of  the 
relationships  between  the  measures.  What  are  the  effects  of  one  measure  on  another,  and  how  can 
we  assure  that  only  the  desired  measure  is  obtained? 

Interpretation  of  measures:  Measures  may  be  quite  judgmental  due  to  imprecise  system 
boundaries,  imposed  constraints,  reference  standards,  and  assumptions. 

Environmental  components:  An  evaluation  requires  the  separation  and  linkage  of  CCIS  and 
users  /  CCIS  and  organization  /  CCIS  and  military  objective. 

Summary  measures:  High  level  measures  have  limitations  in  diagnosis  of  command  and 
control  success  or  failure.  How  do  a  small  number  of  summary  measures  provide  a  comprehensive 
assessment  of  the  complexity  of  a  command  and  control  system? 

Reliability  and  validity:  Assessments  are  often  approached  with  minimal  attention  to 
reliability  and  validity.  The  analyst  must  be  prepared  to  demonstrate  convincingly  the  veracity  of  the 
results  of  an  evaluation. 
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Uncertainty:  Measures  may  be  affected  by  uncertainties  in  the  scenario,  the  model  and  the 
outcome.  C2  decisions  are  made  in  situations  of  degrees  of  imcertainty,  therefore  the  outcome  is  of 
less  significance  in  evaluation  than  in  the  evaluation  of  the  quality  of  the  process. 

Human-in-the-loop:  Variation  in  measurements  due  to  human  involvement  may  well  cause 
unacceptable  levels  of  uncertainty  in  the  results  of  an  evaluation. 

Cost  and  convenience:  Evaluations  of  the  contributions  of  IT  is  labour  intensive.  Measures 
may  be  obtained  in  conjunction  with  Command  Post  Exercises  (CPX),  but  the  formulation  of  CPX 
specifically  for  determination  of  measures  is  rarely  done. 

Training:.  The  effect  of  the  “learning  curve”  on  productivity  in  the  use  of  new  automated 
capabilities  renders  the  object  of  the  evaluation  a  moving  target. 

Modelling:  Improvements  in  the  modelling  of  C2  are  required  in  fidelity  and  resolution  across 
all  hierarchical  levels  of  interest.  The  impact  of  human-in-the-loop,  especially  in  decision  making, 
is  difficult  to  model. 
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7.0  RECOMMENDATIONS 

The  principal  objective  for  MOM  is  to  determine  judgements  of  the  degree  that  C2 
acquisitions  may  improve  force  effectiveness,  and  to  provide  convincing  arguments  for  the 
improvements.  It  is  important  to  stress  that  the  purpose  is  to  assess  the  contributions  of  C2  in  terms 
of  its  effectiveness  on  military  missions,  and  not  on  the  quality  of  the  C2  process  itself  However,  to 
arrive  at  these  assessments,  the  C2  system  must  be  included  in  the  analysis.  The  following  points  are 
required  in  the  assessments: 

a.  the  objectives  of  the  evaluation  must  be  established  and  clearly  stated; 

b.  assirmptions  used  in  the  model  must  be  stated  as  well  as  the  constraints  placed  on  either 
the  evaluation  process  or  the  systems  being  assessed; 

c.  formal  assessments  of  reliability  and  validity  must  be  rmdertaken  in  order  to  place  a  level 
of  confidence  in  the  results  of  the  measurement  process; 

d.  concentrate  effort  on  MOE  and  MOP,  as  they  yield  the  best  insight  into  C3I  effectiveness 
(as  opposed  to  DP  and  MOFE); 

e.  the  capability  to  generate  MOM  should  be  included  as  part  of  system  development;  and 


f  Subject  Matter  Experts  should  be  included  in  studies  of  assessments. 
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8.0  CONCLUSIONS 

No  single  measure  or  methodology  exists  that  satisfactorily  assesses  the  overall  effectiveness 
of  command  and  control  systems;  therefore  a  multi-method,  multi-phase  approach  is  necessary. 

As  a  minimum,  the  following  factors  must  be  considered  in  conducting  an  analysis  of  C2  systems: 

a.  determination  of  the  hierarchy  levels  appropriate  for  the  level  of  analysis  to  establish 
appropriate  MOM; 

b.  identification  of  specific  MOM  that  are  practically  obtainable; 

c.  specification  of  the  means  to  collect  MOM; 

d.  assurance  of  the  validity  and  reliability  of  measures  for  correct  interpretation  with 
quantifiable  levels  of  confidence; 

e.  awareness  that  variation  in  measurements  (e.g.,  due  to  human  involvement)  may  cause 
unacceptable  levels  of  uncertainty,  hence  the  analyst  must  pay  particular  attention  to 
measurements  related  to  the  human  factor; 

f  consideration  that  although  MOFE  may  provide  the  most  persuasive  measures  from  the 
military  perspective,  MOE  and  MOP  are  the  most  meaningful  from  the  OR  analyst 
viewpoint;  and 

g.  accountability  for  the  principles  of  reliability  and  validity  to  avoid  the  risk  of  generating 
false  conclusions. 


Information  Technology  will  continue  to  have  major  impacts  on  military  organization  and 
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operational  concepts.  New  capabilities  of  command  and  control  systems  will  effect  the  underlying 
C2  processes  requiring  further  analysis  to  determine  the  appropriate  measures  of  merit. 

The  domain  of  the  analysis  of  the  effectiveness  of  information  technology  on  Command  and 
Control  remains  difficult.  The  R&D  community  has  made  slow,  but  perceptible  gains  over  the  years; 
the  complexity  of  C2  offers  challenges  for  further  advances. 
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APPENDIX  A  DIVISION  HQ  PERFORMANCE  EVALUATION 


Division  Command  and  Control 
Performance  Evaluation 
Planning  and  Monitoring 


Functional  Elements  HQ  Cells 

Commander 

G1  Personnel 
G2  Intelligence 
G3  Operations 
G4  Supply 

G5  Civil  Military  Cooperation 

Manoeuvre 
Fire  Support 
Engineers 
Tactical  Aviation 
Offensive  Air  Support 
Signals 

Electronic  Warfare 

General  Steps 

Receive  directions  from  higher  headquarters 
Develop  concept  of  operations  and  planning  guidance 
Staff  check  of  resources  and  identify  options  for  courses  of  action 
Develop  outline  plan 
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Develop  detailed  plan,  staff  estimates 
Preparation  and  issue  of  orders 
Execution  of  orders 

Planning  Phase 

-  Develop  planning  guidance 

-  Complete  staff  checks 

-  G1  personnel 

-  G2  intelligence 

-  G3  operations 

-  G4  supply 

-  Artillery 

-  Engineers 

-  Signals 

-  Electronic  Warfare 

-  Tactical  Aviation 

-  Offensive  Air  Support 

-  Produce  outline  plan 

-  Complete  estimate: 

-  G1  personnel 

-  replacements  plan 

-  medical  plan 

-  military  police  plan 

-  G2  intelligence 

-  intelligence  estimate 

-  STAIR  (surveillance,  target  acquisition,  intelligence  and  reconnaissance)  plan 
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-  Counter  STAIR  plan 

-  prepare  meteorological  information 

-  map  acquisition  and  distribution  plan 

-  G3  operations 

-  develop  courses  of  action 

-  prepare  tactical  plan  including  tasks,  priorities,  allocations  of  resources,  movement  plan, 
and  coordination  measures 

-  complete  integration  of  branch  plans  into  operation  plan 

-  G4  supply 

-  determine  requirements,  priorities,  availabilities,  shortfalls 

-  supply  estimates:  personnel  consumption,  ammunition,  POL 

-  transport 

-  maintenance 

-  dumping  program 

-  completion  of  administrative  plan 

-  Artillery 

-  complete  fire  support  plan 

-  allocate  fire  support  resources 

-  defensive  fire  plan 

-  prepare  division  nuclear  sub-package 

-  Engineers 

-  engineer  support  plan  including  task  specifications,  priorities,  and  resource  allocations 

-  mobility  tasks  plan 

-  counter  mobility  tasks  plan 

-  survivability  tasks  plan 

-  Signals 

-  plan  headquarters  location  and  move 

-  command  and  signal  plan 

-  combat  net  radio  plan 
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-  line  plan 

-  dispatch  riders 

-  trunk  system  plan 

-  maintenance  plan 

-  Electronic  Warfare 

-  electronic  warfare  counter  measures  plan 

-  electronic  warfare  support  plan 

-  Tactical  Aviation 

-  aviation  support  plan 

-  attack  helicopter  mission  plan(s) 

-  utility  helicopter  mission  plan(s) 

-  Offensive  Air  Support 

-  offensive  air  support  plan 

-  pre-planned  mission  plan(s) 

-  Completion  of  orders 

-  operation  order 

-  administrative  order 

-  intelligence  order 

-  fire  support  order 

-  engineer  order 

-  signals  order 

-  electronic  warfare  order 

-  aviation  order 

Information  Exchange  in  Planning 


Transactions  by  originator: 
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1 .  to  seek  approval  or  concurrence  from  the  respondent;  for  example,  G4  seeking  approval  from  G3 
for  supply  movement  routes; 

-  to  alter  planning  directions  or  guidance 

-  to  perceived  tasks,  requirements  or  priorities 

-  for  proposed  locations,  routes,  schedules,  etc. 

-  for  operation  order  contribution  -  for  cell's  plan 

2.  to  seek  information  from  the  respondent;  for  example,  G3  seeking  information  on  enemy  situation 
from  G2; 

-  to  clarify  directions  or  plaiming  guidance 

-  on  own  tasks  requirements  or  priorities 

-  on  own  situation,  locations,  strengths 

-  on  availability  of  own  resources  -  on  respondent's  situation,  locations,  strengths 

-  on  availability  of  respondent's  resources 

-  on  content  or  status  of  respondent's  plan 


3.  to  supply  (unsolicited)  information  to  the  respondent;  for  example,  G2  providing  an  intelligence 
briefing  to  the  engineer  advisor. 

-  to  clarify  directions  or  planning  guidance 

-  on  own  tasks,  requirements  or  priorities  -  on  own  situation,  locations,  strengths 

-  on  availability  of  own  resources 

-  as  part  of  regular  briefing  or  report  (e.g  INTSUM) 

-  on  content  or  status  of  own  plan 

-  on  event  or  situation  of  interest  to  respondent 
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Impact  of  transactions: 

Result  or  impact  for  the  originator/respondent 

-  no  impact 

-  caused  delay 

-  prevented  delay  or  speeded  up  work 

-  caused  unnecessary  additional  work 

-  prevented  doing  unnecessary  work 

-  prevented  making  error  in  plan 

-  improved  quality  of  plan 

Importance  of  transactions 

-  critical 

-  useful 

-  could  be  critical 

-  could  be  useful 

-  not  important 
-unknown 

Additional  information  to  develop  flows  of  information 

-  the  originator's  formation  and  cell 

-  the  respondent's  formation  and  cell 

-  the  transaction  medium 

-  the  time  when  the  transaction  was  originated  and  the  time  when  it  was  completed 

-  delays  or  interruptions  in  transactions 
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Factors  for  evaluation  of  the  quality  of  plans 

1.  lack  of  correspondence  with  commander's  intentions; 

2.  tactically  inappropriate  deployment  or  manoeuvre  of  resources; 

3.  inappropriate  definitions  of  tasks; 

4.  inappropriate  priorities; 

5.  underestimated  resource  requirements; 

6.  improper  allocation  of  resources  to  tasks; 

7.  imderestimates  of  capacities  of  allocated  resources  to  meet  task  requirements; 

8.  incorrect  or  unnecessarily  complicated  coordination  requirements; 

9.  incorrect  interpretation  of  intelligence  estimates  of  enemy  capabilities; 

10.  imderestimates  of  difficulties  posed  by  ground  or  terrain; 

1 1.  failure  to  account  for  contingencies; 


(If  a  plan  turns  out  to  need  changes  or  fails,  the  quality  of  plan  assessments  would  make  it  possible 
to  diagnose  whether  or  not  the  changes  were  required  because  of  shortcomings  in  planning  support 
systems  or  because  of  shortcomings  in  the  quality  of  the  plan.) 

Monitoring  Phase 

Objectives. 

-  maintain  an  accurate  and  current  picture  of  the  status  of  operations 

-  provide  commanders  the  means  to  quickly  adjust  plans 

Measures 

-  information  transmitted  in  command  post  networks 
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-  volumes  and  times,  averages  and  variations,  ratios  higher  to  lower  headquarters 
comprehension  of  situation 

-  disposition  of  subordinate  units 

-  nature  of  enemy  activity 

-  sources  of  vulnerability 

-  orders  currently  in  effect 
information  exchange  among  cells 

-  prevented  delay  or  speeded  up  operations 

-  caused  urmecessary  additional  operations 

-  provided  critical  information  on  respondent's  situation 

-  provided  critical  information  on  own  situation 

-  increased  respondent's  knowledge  of  respondent's  situation 

execution  of  plan  alterations 

-  contributing  factors 

-  change  in  higher  formation's  plan 

-  change  in  own  formation's  plan 

-  change  in  lower  formation's  plan 

-  change  in  another  branch's  plan  requiring  change  in  own  plan 

-  unanticipated  losses  of  personnel  or  equipment 

-  unanticipated  shortfalls  in  resources 

-  failure  to  meet  operations  order  schedule 

-  reaction  to  newly  discovered  enemy  actions  or  manoeuvres 

-  unanticipated  ground  or  terrain  obstacles 

-  necessity  for  change  due  to: 

-  the  original  plan  inappropriate 

-  occurrence  of  unanticipated  events 

-  multiple  options  designed  to  create  or  exploit  opportunities 
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time  information  includes: 

-  time  of  event  that  created  need  to  alter  plan 

-  time  of  decision  to  alter  plan 

-  time  when  work  on  altering  plan  started 

-  time  when  plan  alteration  was  completed 

-  time  when  new  order  was  issued 

effort  required  to  produce  the  plan 

-  repeat  entire  planning  cycle 

-  reworked  requirements 

-  redefined  tasks  or  priorities 

-  reworked  resources  quantities  or  allocations 

-  reworked  movements,  locations,  or  timings 

-  made  minor  adjustments 

impact  on  other  operations: 

-  alteration  required  changes  in  plans  at  subordinate  formation 

-  alteration  required  change  in  plans  of  one  or  more  other  branches 

-  alteration  required  change  in  higher  formation's  plan 

expected  outcome  of  plan  alteration: 

-  avoid  failure  to  achieve  main  operation  mission  objectives 

-  avoid  failure  to  meet  one  or  more  of  own  mission  objectives; 

-  prevent  uimecessary  losses  of  personnel  or  equipment; 

-  exploit  imanticipated  opportunity  to  achieve  mission  objectives  sooner  or 
more  efficiently; 

-  exploit  unanticipated  opportunity  to  inflict  losses  on  enemy,  or  thwart 
enemy  operations. 
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